Image processing apparatus and method

ABSTRACT

A surrounding pixel reference unit reads out and stores image data having a predetermined size from a line buffer. A color judgment unit detects the color of the image (reference image) stored in the surrounding pixel reference unit. A parameter determination unit determines parameters used in an individual noise removal unit on the basis of the color detected by the color judgment unit, and supplies them to the individual noise removal unit. The individual noise removal unit reads out, from the line buffer, an image which is located at the same position as that read out by the surrounding pixel reference unit and has a size according to the parameters supplied from the parameter determination unit, and executes a noise removal process for the readout image in accordance with the parameters supplied from the parameter determination unit.

FIELD OF THE INVENTION

The present invention relates to an image processing apparatus andmethod, which can remove noise from an image on which noise issuperposed.

BACKGROUND OF THE INVENTION

Conventionally, a technique for removing noise components from a digitalimage on which noise components that are different from signalcomponents are superposed has been studied. The characteristics of noiseto be removed are diverse depending on their generation factors, andnoise removal methods suited to those characteristics have beenproposed. For example, when an image input device such as a digitalcamera, image scanner, or the like is assumed, noise components areroughly categorized into noise which depends on the input devicecharacteristics of a solid-state image sensing element or the like andinput conditions such as an image sensing mode, scene, or the like, andhas already been superposed on a photoelectrically converted analogoriginal signal, and noise which is superposed via various digitalsignal processes after the analog signal is converted into a digitalsignal via an A/D converter.

As an example of the former, impulse noise that generates an isolatedvalue to have no correlation with surrounding image signal values, noiseresulting from the dark current of the solid-state image-sensingelement, and the like are known. As an example of the latter, noisecomponents are amplified simultaneously with signal components when aspecific density, color, and the like are emphasized in variouscorrection processes such as gamma correction, gain correction forimproving the sensitivity, and the like, thus increasing the noiselevel. As an example of deterioration due to a digital signal process,since an encoding process using a JPEG algorithm extracts a plurality ofblocks from a two-dimensional (2D) image, and executes orthogonaltransformation and quantization for respective blocks, a decoded imagesuffers block distortion that generates steps at the boundaries ofblocks.

In addition to various kinds of noise mentioned above, a factor thatespecially impairs the image quality is noise (to be referred to as“low-frequency noise” hereinafter) which is generated in a low-frequencyrange and is conspicuously observed in an image sensed by a digitalcamera or the like. This low-frequency noise often results from thesensitivity of a CCD or CMOS sensor as a solid-state image sensingelement. In an image sensing scene such as a dark scene with a lowsignal level, a shadowy scene, or the like, low-frequency noise is oftenemphasized due to gain correction that raises signal componentsirrespective of poor S/N ratio. Furthermore, the element sensitivity ofthe solid-state image sensing element depends on its chip area. Hence,in a digital camera which has a large number of pixels within a smallarea, the amount of light per unit pixel consequently decreases, and thesensitivity lowers, thus producing noise. Such low-frequency noise isoften visually recognized as pseudo mottled texture across several toten-odd pixels on a flat portion such as a sheet of blue sky or thelike. Some digital cameras often produce false colors.

As a conventionally proposed noise removal method, a method using amedian filter (to be abbreviated as “MF” hereinafter) which extracts apixel value which assumes a median from those of a pixel of interest andits surrounding pixels, and replaces the pixel value of interest by theextracted value is prevalent.

Also, as a noise removal method effective for impulse noise, blockdistortion mentioned above, and the like, a method using a low-passfilter (to be abbreviated as “LPF” hereinafter) which calculates theweighted mean using the pixel values of a pixel of interest and itssurrounding pixels, and replaces the pixel value of interest by thecalculated weighted mean is used. Furthermore, as a noise removal methodeffective for low-frequency noise, a method of replacing a pixel valueof interest by a pixel value which is probabilistically selected fromthose around the pixel of interest (to be referred to as a “noisedistribution method” hereinafter) has been proposed.

As described above, noise components superposed on an image areinfluenced by various causes. For example, a digital camera suffersmultiple causes, i.e., noise superposed on an analog original signaldepending on, e.g., the input device characteristics of a solid-stateimage sensing element or the like is further amplified by variousdigital image processes executed after the analog original signal isconverted into a digital signal via an A/D converter.

In digital image processes executed by a digital camera, typicalprocesses which are involved in noise generation and amplificationinclude white balance correction, gain correction for improving thesensitivity, saturation correction which is executed for specific colorsto simulate memory colors, and the like.

The white balance correction corrects a phenomenon that an originallywhite image does not appear white due to total color unbalance which iscaused since the amounts of light components that reach an image sensingelement via color filters differ depending on the filter colors, or thenumber of pixels of an image sensing element adopted per pixel of anoutput image differs for respective colors.

The gain correction for improving the sensitivity compensates for aninsufficient amount of light by amplifying signal information of eitheran analog signal obtained from an image sensing element or a digitalsignal after A/D conversion, so as to allow a photographing operationeven when the amount of light in a photographing environment isinsufficient.

The saturation correction corrects, e.g., the saturation of blue tovividly express the color of clear sky, so as to obtain a preferredimage by making colors in an image simulate colors in one's memory.

In the conventional method, flatness in an image is detected, and anoise removal process with a relatively high effect is applied to a flatportion. On the other hand, a noise removal process with a relativelylow effect is applied to an edge portion or a process is skipped so asto minimize adverse effects.

However, as described above, the nature of noise superposed on an imagesignal does hot depend on the flatness of an image. For this reason,with the conventional method, when a noise removal process that cansufficiently remove noise is applied to a given region, other regionssuffer high noise level, and noise cannot be sufficiently removed.Furthermore, the adverse effects of the noise removal process may bevisually conspicuous in other regions.

The present invention has been made in consideration of theaforementioned problems, and has as its object to provide an imageprocessing apparatus and method, which can execute a noise removalprocess more effectively.

SUMMARY OF THE INVENTION

In order to achieve the above object, for example, an image processingapparatus of the present invention comprises the following arrangement.

That is, an image processing apparatus comprising:

judgment means for judging a color of an image for each region having apredetermined size in an image superposed with noise;

determination means for determining a parameter for a noise removalprocess in correspondence with the color determined by the judgmentmeans; and

control means for controlling execution of the noise removal process inaccordance with the parameter determined by the determination means.

In order to achieve the above object, for example, an image processingmethod of the present invention comprises the following arrangement.

That is, an image processing method comprising:

a judgment step of judging a color an image for each region having apredetermined size in an image superposed with noise;

a determination step of determining a parameter for a noise removalprocess in correspondence with the color determined in the judgmentstep; and

a control step of controlling execution of the noise removal process inaccordance with the parameter determined in the determination step.

Other features and advantages of the present invention will be apparentfrom the following description taken in conjunction with theaccompanying drawings, in which like reference characters designate thesame or similar parts throughout the figures thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the functional arrangement of an imageprocessing apparatus according to an embodiment of the presentinvention;

FIG. 2 is a flow chart of a noise removal process executed by an imageprocessing apparatus according to the first embodiment of the presentinvention;

FIG. 3 is a block diagram showing the functional arrangement of anindividual noise removal unit 105 upon executing a noise removal processusing an LPF;

FIG. 4 is a flow chart of a noise removal process executed by the imageprocessing apparatus shown in FIG. 1 when the arrangement shown in FIG.3 is applied to the individual noise removal unit 105;

FIG. 5 is a flow chart showing details of the process for determiningparameters in step S203;

FIG. 6A shows an example of noise superposed on an image;

FIG. 6B show processing ranges of an LPF process, in which two boldframes 602 and 603 indicate LPF processing ranges having a pixel 604 asa pixel of interest;

FIG. 6C shows an image state as a result of an LPF process which isapplied to the image containing noise in FIG. 6A to have the range 602of 5 pixels×5 pixels as a processing range;

FIG. 6D shows an image state as a result of an LPF process which isapplied to the image containing noise in FIG. 6A to have the range 603of 9 pixels×9 pixels as a processing range;

FIG. 7A shows an example of an edge portion present in an image;

FIG. 7B shows an image state as a result of an LPF process which isapplied to the image in FIG. 7A to have the range 602 of 5 pixels×5pixels as a processing range;

FIG. 7C shows an image state as a result of an LPF process which isapplied to the image in FIG. 7A to have the range 603 of 9 pixels×9pixels as a processing range;

FIG. 8A shows an image to be processed;

FIG. 8B shows a change in pixel value upon plotting one line bounded bya bold frame in FIG. 8A along the abscissa;

FIG. 8C shows the first example of weights used upon calculating aweighted mean in an LPF process;

FIG. 8D shows pixel values as a result of an LPF process which isapplied to the image in FIG. 8A using the weights shown in FIG. 8C;

FIG. 8E shows a change in pixel value upon plotting one line bounded bya bold frame in FIG. 8D along the abscissa;

FIG. 8F shows the second example of weights used upon calculating aweighted mean in an LPF process;

FIG. 8G shows pixel values as a result of an LPF process which isapplied to the image in FIG. 8A using the weights shown in FIG. 8F;

FIG. 8H shows a change in pixel value upon plotting one line bounded bya bold frame in FIG. 8G along the abscissa;

FIG. 9 is a block diagram showing the functional arrangement of anindividual noise removal unit which executes a noise removal processusing the noise distribution method, according to the second embodimentof the present invention;

FIG. 10 is a flow chart of a noise removal process executed by an imageprocessing apparatus according to the second embodiment of the presentinvention, when the individual noise removal unit according to thesecond embodiment of the present invention is used as the individualnoise removal unit 105;

FIG. 11A shows an image state as a result of a noise distributionprocess which is applied to the image in FIG. 6A to have the range 602of 5 pixels×5 pixels as a processing range;

FIG. 11B shows an image state as a result of a noise distributionprocess which is applied to the image in FIG. 6A to have the range 603of 9 pixels×9 pixels as a processing range;

FIG. 12A shows an image state as a result of a noise distributionprocess which is applied to the image in FIG. 7A to have the range 602of 5 pixels×5 pixels as a processing range;

FIG. 12B shows an image state as a result of a noise distributionprocess which is applied to the image in FIG. 7A to have the range 603of 9 pixels×9 pixels as a processing range;

FIG. 13A shows an image to be processed;

FIG. 13B shows a change in pixel value upon plotting one line bounded bya bold frame in FIG. 13A along the abscissa;

FIG. 13C shows an example of a formula that represents a pixel valuesubstitution condition used in the noise distribution process;

FIG. 13D shows a result of the noise distribution process applied toFIG. 13A as the image to be processed on the basis of the formula inFIG. 13C;

FIG. 13E shows pixel values in one line bounded by a bold frame in FIG.13D;

FIG. 13F shows another example of a formula that represents a pixelvalue substitution condition used in the noise distribution process;

FIG. 13G shows a result of the noise distribution process applied toFIG. 13A as the image to be processed on the basis of the formula inFIG. 13F;

FIG. 13H shows pixel values in one line bounded by a bold frame in FIG.13G;

FIG. 14 is a block diagram showing the functional arrangement of anindividual noise removal unit that executes a noise removal processusing an MF, according to the third embodiment of the present invention;

FIG. 15 is a flow chart of a noise removal process executed by an imageprocessing apparatus according to the third embodiment of the presentinvention, when the individual noise removal unit according to the thirdembodiment of the present invention is used as the individual noiseremoval unit 105;

FIG. 16 is a block diagram showing the functional arrangement of animage processing apparatus which combines noise removal processes,according to the fourth embodiment of the present invention;

FIG. 17A shows an image on which noise components are superposed;

FIG. 17B shows a result of the noise distribution process of a mainnoise removal unit 1602, which is applied to the image in FIG. 17A;

FIG. 17C shows a result of the MF process of a front-end sub noiseremoval unit 1601, which is applied to the image in FIG. 17A;

FIG. 17D shows a result of the noise distribution process which isfurther applied to FIG. 17C;

FIG. 18 is a block diagram showing the functional arrangement of animage processing apparatus which combines noise removal processes,according to the fifth embodiment of the present invention;

FIG. 19A shows an image on which noise components are superposed;

FIG. 19B shows a result of the noise distribution process of a mainnoise removal unit 1801, which is applied to the image in FIG. 19A;

FIG. 19C shows a result of the LPF process of a back-end sub noiseremoval unit 1802, which is further applied to the image in FIG. 19B;and

FIG. 20 is a block diagram showing the functional arrangement of animage processing apparatus according to the sixth embodiment of thepresent invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

In embodiments to be described hereinafter, each process for reducingnoise from an image on which noise is superposed is called a “noiseremoval process”, and this process has a purpose of not completelyremoving noise from an image but reducing noise to a visuallyimperceptible level.

First Embodiment

FIG. 1 is a block diagram showing the functional arrangement of an imageprocessing apparatus according to this embodiment. The image processingapparatus shown in FIG. 1 comprises an input terminal 100, line buffer101, surrounding pixel reference unit 102, color judgment unit 103,parameter determination unit 104, individual noise removal unit 105, andoutput terminal 106.

The input terminal 100 receives color image data (to be referred to asimage data simply or as original image data hereinafter) formed of R, G,and B color signals, and outputs the image data to the line buffer 101to be described below. The line buffer 101 stores and holds the inputimage data for respective lines. The surrounding pixel reference unit102 comprises line buffers for several lines, and stores image data fora predetermined size sequentially read out from the image data stored inthe line buffer 101.

The color judgment unit 103 refers to image data held by the surroundingpixel reference unit 102 to detect an image color indicated by thatimage data. The color detection method will be described later. Theparameter determination unit 104 determines parameters used in a noiseremoval process to be executed by the next individual noise removal unit105 on the basis of the detection result of the color judgment unit 103,and supplies the determined parameters to the individual noise removalunit 105. The individual noise removal unit 105 can execute a noiseremoval process using a noise removal method such as an MF method, LPFmethod, noise distribution method, or the like, and applies the noiseremoval process to the image supplied from the line buffer 101 inaccordance with the parameters supplied from the parameter determinationunit 104. The image that has undergone the noise removal process isexternally output via the output terminal 106.

The noise removal process executed by the image processing apparatuswith the above arrangement will be described below with reference toFIG. 2 which is a flow chart of that process.

Note that a computer may function as an image processing apparatusaccording to this embodiment by loading an image processing program as aprogram group which implements the functions of the respective unitsshown in FIG. 1, and executing the noise removal process. In this case,FIG. 1 shows the functional arrangement of the image processing program.Also, in this case, FIG. 2 shows a flow chart of the process that theimage processing program makes the computer execute. As the sizes of animage to be input to the image processing apparatus, the number ofhorizontal pixels is represented by Width, and the number of verticalpixels is represented by Height.

Variable i which indicates the vertical address of each pixel that formsan input image, and variable j that indicates its horizontal address arerespectively reset to zero (steps S200 and S201). The surrounding pixelreference unit 102 reads out image data (reference image data) having apredetermined size to have a position (j, i) as the center from the linebuffer 101, and stores the readout data. At the same time, the colorjudgment unit 103 detects the color of the reference image (step S202)As a color detection method, for example, the average densities ofrespective colors in pixels included in the reference image arecalculated, and the color of the reference image is detected on thebasis of a combination of the average densities of respective colors.

When the reference image includes an edge (the boundary betweendifferent color regions), a color defined by the combination does notoften normally reflect the color of the reference image. In such case,the reference image may be further divided into sub-images to avoidrespective color regions from being included in one image, and eachdivided sub-image may undergo the color determination. In this case, thecolor determination unit 103 sends a message indicating that thereference image is further divided into sub-images to the subsequentparameter determination unit 104 and individual noise removal unit 105.

The parameter determination unit 104 determines parameters used in theindividual noise removal unit 105 on the basis of the color detected bythe color judgment unit 103, and supplies them to the next individualnoise removal unit 105 (step S203). When the reference image is furtherdivided into sub-images in step S202, as described above, the parameterdetermination unit 104 determines parameters for each divided sub-image,and supplies the determined parameters to the individual noise removalunit 105.

The individual noise removal unit 105 reads out an image (image to beprocessed), which is located at the same position (the same imagecentral position) as the reference image and has a size according to theparameters supplied from the parameter determination unit 104, from theline buffer 101. The unit 105 then executes a noise removal process ofthe readout image according to the parameters supplied from theparameter determination unit 104 using predetermined one of noiseremoval methods such as an MF method, LPF method, noise distributionmethod, and the like (step S204). When the reference image is furtherdivided into sub-images in step S202, as described above, the individualnoise removal unit 105 executes a noise removal process for each dividedsub-image.

The horizontal address j is counted up by 1 to move the position of theimage to be processed one pixel to the right (step S205), and it is thenchecked if the address j is smaller than Width (step S.206). Theprocesses in steps S202 to S205 are repeated as long as j<Width.

On the other hand, if j≧Width, the flow advances from step S206 to stepS207, and the vertical address i is counted up by one to move theposition of the image to be processed one pixel downward (step S207). Itis then checked if the address i is smaller than Height (step S208) Theprocesses in steps S201 to S207 are repeated as long as i<Height. Ifi≧Height, the aforementioned processes end.

The individual noise removal unit 105 will be described below. FIG. 3shows the functional arrangement of the individual noise removal unit105 when a noise removal process is executed using the aforementionedLPF method. Note that the same reference numerals in FIG. 3 denote thesame parts as those in FIG. 1, and a description thereof will beomitted.

The individual noise removal unit 105 shown in FIG. 3 comprises inputterminals 300, 301, and 303, surrounding pixel reference unit 302, andfiltering product sum calculation unit 304. The input terminal 300receives an image from the line buffer 301. The input terminal 301receives information which is included in the parameters supplied fromthe parameter determination unit 104 and indicates the size of a regionthat is to undergo a noise removal process. The input terminal 303receives the parameters which are supplied from the parameterdetermination unit 104 and are used in a filtering process. Thesurrounding pixel reference unit 302 stores an image having a sizeaccording to the parameter supplied from the parameter determinationunit 104 in the image (an image at the same position as the referenceimage in the original image) input via the input terminal 300.

The filtering product sum calculation unit 304 executes a process forcalculating a weighted mean of the pixel values of the image held by thesurrounding pixel reference unit 302 using the parameters input via theinput terminal 303, and replacing the pixel value of the pixel ofinterest by the calculated weighted mean.

FIG. 4 is a flow chart of a noise removal process executed by the imageprocessing apparatus shown in FIG. 1 when the arrangement shown in FIG.3 is applied to the individual noise removal unit 105. The noise removalprocess will be described below using the flow chart in FIG. 4. Notethat ranges bounded by the broken lines in FIG. 4 respectively indicatethe processes executed in steps S203 and S204, and the processes otherthan steps S203 and S204 are not shown in FIG. 4. That is, FIG. 4 is aflow chart showing details of the parameter determination process instep S203, and the noise removal process in step S204.

The size of an image to be processed (calculation range) is obtainedbased on the color detection result of the color judgment unit 103 (stepS400). Note that sizes which are experimentally determined in advance inaccordance with effects or adverse effects to be described later areheld in a table or the like as calculation ranges, and are selected onthe basis of the color detection result. In the following description,let Area_(Width) be the horizontal size (the number of pixels) of thecalculation range, and Area_(Height) be the vertical size (the number ofpixels).

Then, weighting coefficients A(x, y) used upon calculating a weightedmean using the pixel values of pixels within the calculation range aredetermined (step S401). Note that respective weighting coefficients A(x,y) correspond to pixels which form the image in the calculation range.Values which are experimentally determined in advance in accordance witheffects or adverse effects to be described later are held in a table orthe like as weighting coefficients A(x, y), and are selected on thebasis of the color detection result. With the processes in steps S400and S401, the parameter determination process is implemented.

The noise removal process is then executed. All variables Sum_(R),Sum_(G), and Sum_(B) which correspond to R, G, and B in respectivepixels in the calculation range and are used in subsequent processes arereset to zero (step S402).

Variable ii that indicates the vertical processing address of each pixelwhich forms the image within the calculation range is reset to zero(step S403). Likewise, variable jj that indicates the horizontalprocessing address of each pixel which forms the image within thecalculation range is reset to zero (step S404).

In each pixel which forms the image within the calculation range,product sum calculations of I_(R), I_(G), and I_(B) indicating R, G, andB pixel values and weighting coefficient A are made to obtain R, G, andB cumulative sum values (step S405). Then, variable jj indicating thehorizontal address is counted up by one to move the pixel position whichis to undergo product sum calculations one pixel to the right (stepS406). The processes in steps S405 and S406 are repeated untiljj≧Area_(Width) (step S407).

Variable ii indicating the vertical address is counted up by one to movethe pixel position which is to undergo product sum calculations onepixel downward (step S408). The processes in steps S404 to S408 arerepeated until ii≧Area_(Height).

With the above processes, product sum values Sum_(R), Sum_(G), andSum_(B) of all pixels which form the image in the calculation range andweighting coefficients can be obtained. Then, the average values ofthese product sum values Sum_(R), Sum_(G), and Sum_(B) in the imagewithin the calculation range are calculated (step S410). That is, eachproduct sum value is divided by a sum total S_(w)=Σ_(y)Σ_(x)A (x, y) ofweights (where Σ_(a)f(a) is the sum total of f(a) of all “a”s.

The respective quotients replace the R, G, and B pixel values of thepixel of interest as values Fr(j, i), Fg(j, i), and Fb(j, i).

In the above process, identical weighting coefficients are used for R,G, and B components. Alternatively, different weighting coefficients maybe generated for respective components. In such case, the weighted meancalculation process in steps S402 to S410 is executed using individualweights for an individual calculation range of each component.

In the description of the above process, an image is made up of R, G,and B components. However, the above process may be applied to luminanceand color difference components used in JPEG or the like, orcomplementary color components such as C, M, Y, and K components or thelike used as ink colors in a printer or the like.

The detailed process for determining parameters in step S203 will bedescribed below with reference to FIG. 5 which is a flow chart of thatprocess. This process may be applied to the-process for obtainingweighting coefficients in step S401.

If it is determined as a result of color detection of the color judgmentunit 103 that the region of interest is a dark portion (step S500), theflow advances to step S501 to set parameters for the dark portion (stepS501). On the other hand, if it is determined that the color of theregion of interest is flesh color (step S502), the flow advances to stepS503 to set parameters for flesh color. Also, if it is determined thatthe color of the region of interest is blue (step S504), the flowadvances to step S505 to set parameters for blue (step S505). If it isdetermined that the color of the region of interest is none of the abovecolors, the flow advances to step S506 to set parameters for othercolors (step S506). Upon completion of the aforementioned processes, theflow advances to step S204.

Note that the detection process is executed in the order of darkportion→flesh color→blue. However, the present invention is not limitedto such specific order, and is not limited to these colors.

The color of the region of interest, and the effect and adverse effectof the noise removal process will be explained below. FIG. 6A shows anexample of noise superposed on an image. A hatched pixel region 600includes pixels that contain noise components, and is present in animage as a cluster of several to ten-odd successive pixels. A whitepixel region 601 corresponds to that other than noise components.

FIG. 6B shows the processing range of the LPF process. Regions 602 and603 within two bold frames indicate LPF processing ranges which have apixel 604 as a pixel of interest. Note that the region 602 indicates a5×5 pixel processing range, and the region 603 indicates a 9×9 pixel,processing range.

FIG. 6C shows the state of an image as a result of the LPF process whichis applied to the image containing noise shown in FIG. 6A using the 5×5pixel range 602 as the processing range. A black pixel region 605indicates a region where noise components are attenuated. Also, a graypixel region 606 indicates a region where noise components are diffusedby the LPF process although it is originally free from any noisecomponents. A hatched pixel region 607 indicates a region where theeffect of the processing is small due to an insufficient processingrange size compared to the noise range. In the region 607, since the LPFprocess is done within the noise range, the weighted mean of noisecomponents is calculated, and the effect of attenuating noise componentsis small. On the other hand, in the region 605, since the weighted meanis calculated using the pixel values of the region 601 which isoriginally free from any noise component, noise components areattenuated.

FIG. 6D shows the state of an image as a result of the LPF process whichis applied to the image containing noise shown in FIG. 6A using the 9×9pixel range 603 as the processing range. In FIG. 6D, since the LPFprocessing range is sufficiently broader than the noise range, no regionwhere the effect of the process is small is present unlike in FIG. 6C.As shown in FIGS. 6A, 6B, 6C, and 6D, the noise removal effect can beimproved by assuring a larger processing range.

FIG. 7A shows an example of an edge portion present in imageinformation. Reference numeral 700 denotes a low-density region (aregion indicated by white pixels in FIG. 7A); and 701, a high-densityregion (a region indicated by gray pixels in FIG. 7A). In the followingdescription, the processing ranges 602 and 603 of two sizes shown inFIG. 6B are used.

FIG. 7B shows the state of an image as a result obtained of the LPFprocess is applied to the image of FIG. 7A using the 5×5 pixel range 602as the processing range. A region 702 (hatched pixel region in FIG. 7B)indicates a region, the density values of which belonged to thelow-density region 700 before the process, but which increase due todiffusion of the pixel values of the high-density region as a result ofthe LPF process. A region 703 (black pixel region in FIG. 7B) indicatesa region, the density value of which belong to the high-density region701 before process but decrease due to diffusion of pixel values as aresult of the LPF process.

FIG. 7C shows the state of an image as a result obtained of the LPFprocess is applied to the image of FIG. 7A using the 9×9 pixel range 603as the processing range. In FIG. 7C, the ranges of the regions 702 and703 where the pixel values are diffused broaden compared to FIG. 7B. Theregion 702 or 703 is visually recognized as a blur. For this reason, theregion 702 or 703 is preferably narrower since it becomes harder tovisually recognize as an adverse effect.

FIG. 8A shows an image to be processed. In FIG. 8A, reference numeral800 denotes an isolated region in an image (a region having pixel valueswhich are largely different from those of surrounding pixels); and 801,a non-isolated region. FIG. 8B is a graph showing a change in pixelvalue. In FIG. 8B, the abscissa plots one line bounded by the bold framein FIG. 8A, and the ordinate plots the pixel values of pixels.

FIG. 8C shows the first example of weights used upon calculating theweighted mean in the LPF process, i.e., an example in which the pixel ofinterest has a large weight. FIG. 8D shows pixel values as a result ofthe LPF process which is applied to the image of FIG. 8A using theweights shown in FIG. 8C. FIG. 8E is a graph showing a change in pixelvalue. In FIG. 8E, the abscissa plots one line bounded by the bold framein FIG. 8D, and the ordinate plots the pixel values of pixels.

FIG. 8F shows the second example of weights used upon calculating theweighted mean in the LPF process, i.e., an example in which the pixel ofinterest has a small weight. FIG. 8G shows pixel values as a result ofthe LPF process which is applied to the image of FIG. 8A using theweights shown in FIG. 8F. FIG. 8H is a graph showing a change in pixelvalue. In FIG. 8H, the abscissa plots one line bounded by the bold framein FIG. 8G, and the ordinate plots the pixel values of pixels.

Upon comparison between FIGS. 8E and 8H as the results of two differentLPF processes, the pixel values of pixels which neighbor the isolatedregion slightly increase, and those of the isolated region decreaseslightly in the result shown in FIG. 8E. However, the width of the mostisolated portion near the isolated region is w1 as in the image shown inFIG. 8A, and the image signal suffers less deterioration. For thisreason, if the isolated region shown in FIG. 8B is an image as a unitwhich is to undergo noise removal, the adverse effect is relativelysmall.

On the other hand, in the result shown in FIG. 8H, the pixel values ofpixels which neighbor the isolated region increase largely, while thoseof the isolated region decrease largely. Also, the width of the mostisolated portion near the isolated region is w2 unlike in the imageshown in FIG. 8A, and the large effect of the process appears. For thisreason, if the isolated region in FIG. 8B corresponds to noisecomponents, the processing effect is observed.

The noise removal effects and adverse effects with respect to theprocessing ranges and weights in the LPF process have been explainedusing FIGS. 6A, 6B, 6C, 6D, 7A, 7B, 7C, 8A, 8B, 8C, 8D, 8E, 8F, 8G, and8H. On the other hand, the strength of noise, and visual conspicuity ofthe adverse effects after the process have different natures dependingon the color of the processing region.

Noise shown in FIGS. 6A, 6B, 6C, and 6D, or noise like the isolatedregion in FIGS. 8A, 8B, 8C, 8D, 8E, 8F, 8G, and 8H especially becomesconspicuous when an image on which noise has already been superposed ina digital camera is amplified by a process such as luminance gainadjustment, saturation gain adjustment, or the like. A digital camera,which uses primary color filters, is designed to set a lower gain of animage signal after image sensing for a G component that has a highercontribution rate to a visually conspicuous luminance component among R,G, and B color components, since the amount of light to be obtained isincreased by using a larger number of G filters than other colorfilters. However, as for Rand B components, an image signal undergoes anamplification process to compensate for insufficient amounts of light.

Furthermore, in order to simulate a preferred color so as to vividlyexpress, e.g., clear sky, especially, an R component often undergoeslarge gain adjustment. If large gain adjustment is made, noisecomponents are amplified at the same time. For this reason, many noisecomponents are often superposed on a blue region represented by clearsky or the like. Also, a dark portion of a night scene, shadowy scene,or the like with an insufficient amount of light undergoes noiseamplification by gain adjustment, and many noise components are oftensuperposed on such portion. Conversely, depending on the characteristicsof an image input/output device, a pseudo edge is readily generated in adark portion since overall pixel values are raised as a result of gainadjustment. As a result, an adverse effect often becomes conspicuousnear the pseudo edge.

For the blue region on which many noise components are more likely to besuperposed, the effect of the noise removal process must be increased toremove noise. As shown in FIGS. 6A, 6B, 6C, and 6D, the processing rangeis broadened for the blue range to increase the effect of the noiseremoval process. Also, weight parameters are changed, as shown in FIG.8H, to enhance the influence of the process, thus allowing an effectivenoise removal process.

On the other hand, a person is visually very sensitive to a change inone's face. A person often sensitively detects even slight changes infine wrinkles, three-dimensional patterns on a skin surface, and thelike, and feels odd. Since the adverse effects are preferably minimizedfor a human face, a small size of the processing range is adopted for aflesh color region to reduce the adverse effects, as shown in theexample of FIGS. 7A, 7B, and 7C. Also, weight parameters are changed, asshown in FIG. 8E, to reduce the influence of the process, thus allowinga noise removal process which suffers less visual adverse effects.

In this embodiment, the calculation range and weighting coefficientshave been discussed as parameters to be changed. However, the presentinvention is not limited to those specific parameters.

As described above, the noise removal process according to thisembodiment can determine a region with many noise components, and aregion where adverse effects of the noise removal process are visuallyconspicuous, on the basis of the color around the processing region. Asa result, the effects of the noise removal process can be improved whilesuppressing the adverse effects of the noise removal process.

Second Embodiment

The first embodiment has exemplified a case wherein the noise removaleffects are improved while suppressing adverse effects, by changingprocessing parameters in correspondence with the color of the region ofinterest, upon executing the noise removal process using the LPFprocess. This embodiment will exemplify an effective noise removalprocess by changing processing parameters in correspondence with thecolor of the region of interest when another noise removal method isused.

An image processing apparatus according to this embodiment issubstantially the same as the image processing apparatus according tothe first embodiment, except for the arrangement of the individual noiseremoval unit 105 and parameters supplied from the parameterdetermination unit 104.

FIG. 9 is a block diagram showing the functional arrangement of theindividual noise removal unit 105 that executes a noise removal processusing the noise distribution method. The same reference numerals denotethe same parts as those in FIG. 3, and a description thereof will beomitted. The individual noise removal unit shown in FIG. 9 comprisesinput terminals 300, 301, 900, 903, surrounding pixel reference unit302, pixel value selector 901, random number generator 902, and pixelvalue determination unit 904.

The input terminal 900 receives parameters which are supplied from theparameter determination unit 104 and are used in the pixel valuedetermination unit 904. The pixel value selector 901 arbitrarily selectsa pixel from the surrounding pixel reference unit 302 on the basis of apseudo random number generated by the random number generator 902, andparameters input via the terminal 900, and reads out its pixel value.The input terminal 903 receives parameters which are supplied from theparameter determination unit 104 and are used in the pixel valuedetermination unit 904.

The pixel value determination unit 904 calculates and determines thepixel value of a pixel of interest after a noise removal process usingthe parameters input via the input terminal 903, the pixel value of thepixel of interest, which is to undergo the noise removal process and isread out from the surrounding pixel reference unit 302, and the pixelvalue of the pixel selected by the pixel value selector 901.

FIG. 10 is a flow chart of a noise removal process executed by the imageprocessing apparatus according to this embodiment, when the individualnoise removal unit according to this embodiment with the abovearrangement is used as the individual noise removal unit 105. The noiseremoval process will be described below using the flow chart of FIG. 10.Note that ranges bounded by the broken lines in FIG. 10 respectivelyindicate the processes executed in steps S203 and S204, and theprocesses other than steps S203 and S204 are not shown in FIG. 10. Thatis, FIG. 10 is a flow chart showing details of the parameterdetermination process in step S203, and the noise removal process instep S204.

As in step S400, the parameter determination unit 104 obtains the sizeof an image to be processed (calculation range) (step S1000). In thefollowing description, let Area_(Width) be the horizontal size of thecalculation range, and Area_(Height) be its vertical size. Also, theparameter determination unit 104 determines threshold values Thr, Thg,and Thb used in subsequent processes for R, G, and B components on thebasis of the color detection result of the color judgment-unit 103 (stepS1001). The random number generator 902 generates a pseudo random number(step S1002) The pixel value selector 901 determines the values ofvariables a and b which indicate the horizontal and vertical relativepositions from the pixel of interest, on the basis of the generatedrandom number and the color detection result of the color judgment unit103 (step S1003). As a determination method of a and b, for example, therandom number generator 902 generates a pseudo random number.

Note that the values of variables a and b are determined not to exceedthe size of the calculation range obtained in step S1000. For example,if the size of the calculation range is 9×9 pixels having the pixel ofinterest as the center, values a and b are set using a remaindercalculation based on the generated random number to fall within theranges −4≦a≦4 and −4≦b≦4.

With the processes in steps S1000 to S1003, parameters used in thesubsequent processes can be determined.

A noise removal process using these parameters is executed. In the noiseremoval process, the following checking processes are made (step S1004)to see whether or not:|Ir(j, i)−Ir(j+b, i+a)|<Thr and|Ig(j, i)−Ig(j+b, i+a)|<Thg and|Ib(j, i)−Ib(j+b, i+a)|<Thbwhere Ir(j, i) is the pixel value of the R component, Ig(j, i) is thepixel value of the G component, and Ib(j, i) is the pixel value of the Bcomponent all of the pixel of interest located at a coordinate position(j, i). Also, |x| is the absolute value of x.

That is, the above checking processes check whether or not the absolutevalues of the differences between three, R, G, and B component values ofa selected pixel arbitrarily selected from the calculation range, andthose of the pixel of interest of become smaller than the thresholdvalues. If all the three, R, G, and B components are smaller than thethreshold values as a result of the above checking processes, the flowadvances to step S1005, and the pixel values of the pixel of interestare updated by those of the selected pixel (step S1005). On the otherhand, if not all the three, R, G, and B components are smaller than thethreshold values, the flow advances to step S1006, and the pixel valuesof the pixel of interest are not updated.

Note that the parameters determined in step S1000 may have differentvalues for R, G, and B components. In such case, the noise distributionprocess in steps S1002 to S1006 is executed using individual thresholdvalues for individual calculation ranges for respective colors.

In the description of the above process, an image is made up of R, G,and B components. However, the above process may be applied to luminanceand color difference components used in JPEG or the like, orcomplementary color components such as C, M, Y, and K components or thelike used as ink colors in a printer or the like. Also, the detailedprocess for determining parameters in step S203 follows the flow chartshown in FIG. 5, as in the first embodiment.

The color of the region of interest, and the effects and adverse effectsof the noise removal process will be described below. FIGS. 11A and 11Bshow the relationship between noise generated in the image shown in FIG.6A, and the results of the noise distribution process. Also, the sizesof the regions 602 and 603 in FIG. 6B are used as those of thecalculation range.

FIG. 11A shows an image state as a result of the noise distributionprocess which is applied to the image of FIG. 6A using the 5×5 pixelregion 602 as the processing range. Referring to FIG. 11A, referencenumeral 1100 denotes pixels, which belonged to the non-noise region 601before the process, but to which the pixel values of the noise regionare distributed as a result of the noise distribution process. Referencenumeral 1101 denotes *pixels, which belonged to the noise region 600before the process, but which are replaced by the pixel values of thenon-noise region 601 since the noise region is distributed as a resultof the noise distribution process.

In FIG. 11A, since the processing region is smaller than the size of thenoise region 600, the central portion of the noise region 600 undergoesa pixel substitution process within the noise region 600, and theobtained noise removal effect is insufficient. On the other hand, FIG.11B shows an image state as a result of the noise distribution processwhich is applied to the image of FIG. 6A using the 9×9 pixel range 603as the processing range.

In FIG. 11B, the process is done using the processing region which islarge enough with respect to the size of the noise region 600. For thisreason, the central portion of the noise region 600 undergoes pixelvalue substitution, and a cluster of noise components, which are readilyvisually detected, are distributed, thus obtaining a noise removaleffect.

FIGS. 12A and 12B show the relationship between an edge portion presentin the image shown in FIG. 7A, and the results of the noise distributionprocess. In the following description, the sizes of the regions 602 and603 in FIG. 6B are used as those of the calculation range. FIG. 12Ashows an image state as a result of the noise distribution process whichis applied to the image of FIG. 7A using the 5×5 pixel range 602 as theprocessing range.

In FIG. 12A, reference numeral 1200 denote pixels, which belonged to thelow-density region 700 before the process, but to which the pixel valuesof the high-density region 701 are distributed as a result of theprocess. Reference numeral 1201 denotes pixels, which belonged to thehigh-density region 701 before the process, but to which the pixelvalues of the low-density region 700 are distributed as a result of theprocess. In FIG. 12A, since the processing range is relatively small,pixels are distributed only near the edge in the image region shown inFIG. 7A.

On the other hand, FIG. 12B shows an image state as a result of thenoise distribution process which is applied to the image in FIG. 7Ausing the 9×9 pixel region 603 as the processing range. In FIG. 12B,since the processing range size is large, pixels are distributed even topixels farther from the edge in the image region shown in FIG. 7A acrossthe edge boundary. When pixels near the edge boundary are distributedover a broad range, the edge blurs, resulting in deterioration of theimage quality. When the large processing range shown in FIG. 12B isapplied to the image edge portion shown in FIG. 7A, the adverse effectof the process becomes visually conspicuous.

FIG. 13A shows an image to be processed. Reference numeral 1300 denotesan image region which has pixel values (pixel values=25 in FIG. 13A)different from those of a surrounding region. Reference numeral 1301denotes a pixel region around the pixel region 1300, which has pixelvalues=20 as an example. Reference numeral 1302 denotes a pixel regionaround the pixel region 1300, which has pixel values=15 as an example.FIG. 13B is a graph showing a change in pixel value. In FIG. 13B, theabscissa plots one line bounded by the bold frame in FIG. 13A, and theordinate plots the pixel values of pixels.

FIG. 13C shows an example of a formula that expresses a pixel valuesubstitution condition used in the noise distribution process. In FIG.13C, P_(org) is the pixel value of a pixel of interest, and P_(sel) isthat of a selected pixel. In the formula of FIG. 13C, threshold valueTh1=8, and this means that a pixel value is substituted if the absolutevalue of the difference between the pixel values of the pixel ofinterest and the selected pixel is equal to or smaller than 8.

FIG. 13D shows the result of the noise distribution process which isapplied to FIG. 13A as the image to be processed on the basis of theformula in FIG. 13C. Referring to FIG. 13D, since the absolute values ofthe differences between the pixel values of pixels in the regions 1300and 1301 in FIG. 13A become equal to or smaller than 8, pixel values aresubstituted. On the other hand, since the absolute values of thedifferences between the pixel values of pixels in the regions 1300 and1302 become 10, no pixel value substitution is made. FIG. 13E showspixel values in one line bounded by the bold frame in FIG. 13D.

FIG. 13F shows an example of another formula that expresses a pixelvalue substitution condition used in the noise distribution process. InFIG. 13F, threshold value Th2=12 unlike in FIG. 13C. FIG. 13G shows theresult of the noise distribution process applied to FIG. 13A as theimage to be processed on the basis of the formula in FIG. 13F. Referringto FIG. 13G, since not only the absolute values of the differencesbetween pixel values of pixels in the regions 1300 and 1301 in FIG. 13Abecome equal to or smaller than 8, and the absolute values of thedifferences between pixel values of pixels in the regions 1300 and 1302become 10, i.e., the absolute values of the differences between pixelvalues also become smaller than the threshold value, pixel values aresubstituted. FIG. 13H shows pixel values in one line bounded by the boldframe in FIG. 13G.

Upon comparison between FIGS. 13E and 13H as the results of the twodifferent noise distribution processes, pixels on the left side of thegraph in FIG. 13E undergo pixel value substitution, but those on theright side of the graph do not undergo pixel value substitution. Thatis, the shape of the image to be processed near the regions 1300 and1302, which have the same condition as that on the right side of thegraph, remains unchanged. When the region 1300 is an image, an image asa result of the noise removal process is preferably less modified. Incase of the example shown in FIG. 13D or 13E, the shape of the region1300 is not completely disturbed, and the adverse effect can besuppressed.

On the other hand, in FIG. 13H, pixel values are substitutedirrespective of their positions along the abscissa, and the shape of theregion 1300 is disturbed. For this reason, when the region 1300corresponds to noise components, the processing effect is visible.

The noise removal effects and adverse effects with respect to theprocessing ranges and threshold values in the noise distribution processhave been explained using FIGS. 11A and 11B, FIGS. 12A and 12B, andFIGS. 13A, 13B, 13C, 13D, 13E, 13F, 13G, and 13H. As has been explainedin the first embodiment, the strength of noise, and visual conspicuityof the adverse effects after the process have different naturesdepending on the color of the processing region. For this reason, inthis embodiment as well, the noise removal effect can be improved bysetting parameters which broaden the processing range, those which sethigher selection probabilities of pixel values for outer pixels withinthe processing region, those which set a higher threshold value, and soforth, when the processing region has a color with high noise level. Onthe other hand, the adverse effects due to the noise removal process canbe suppressed by setting parameters which narrow down the processingrange, those which set higher selection probabilities of pixel valueswithin the processing region for pixels closer to the pixel of interest,those which set a lower threshold value, and so forth, when theprocessing region has a color that emphasizes the adverse effects.

Note that this embodiment has exemplified the calculation ranges andthreshold values as parameters to be changed. However, the presentinvention is not limited to such specific parameters.

As described above, the noise removal process according to thisembodiment can determine a region with many noise components, and aregion where adverse effects of the noise removal process would bevisually conspicuous on the basis of the color around the processingregion. As a result, the effects of the noise removal process can beimproved while suppressing the adverse effects of the noise removalprocess.

Third Embodiment

The first embodiment described above has exemplified a case wherein thenoise removal process which can enhance its effect and can suppress itsadverse effect is implemented by determining the processing ranges andweights used upon calculating the weighted mean on the basis ofconspicuity of noise, and that of the adverse effect of the noiseremoval process depending on the color of the region of interest in thenoise removal process using an LPF.

The present invention can be applied to an MF method that has beendescribed above as a typical noise removal processing method in additionto the methods exemplified in the first and second embodiments. In theMF method, a median pixel value is selected from all pixel values withinthe processing range, and substitutes the pixel value of the pixel ofinterest. This method is particularly effective for spot noise which hasvery low correlation with surrounding pixels. In this embodiment aswell, the same reference numerals denote the same items as thosedescribed above, and a description thereof will be omitted.

FIG. 14 is a block diagram showing the functional arrangement of anindividual noise removal unit which executes a noise removal processusing an MF according to this embodiment. The same reference numerals inFIG. 14 denote the same parts as those in FIG. 3, and a descriptionthereof will be omitted. The individual noise removal unit shown in FIG.14 comprises input terminals 300 and 301, surrounding pixel referenceunit 302, and median acquisition unit 1400. The individual noise removalunit with the arrangement of FIG. 14 executes a process for acquiring amedian pixel value from all pixels within the processing range, andsubstituting this median pixel value as a new pixel value of the pixelof interest.

FIG. 15 is a flow chart of a noise removal process executed by an imageprocessing apparatus according to this embodiment, when the individualnoise removal unit according to this embodiment is used as theindividual noise removal unit 105. The noise removal process will bedescribed below using the flow chart of FIG. 15. Note that rangesbounded by the broken lines in FIG. 15 respectively indicate theprocesses executed in steps S203 and S204, and the processes other thansteps S203 and S204 are not shown in FIG. 15. That is, FIG. 15 is a flowchart showing details of the parameter determination process in stepS203, and the noise removal process in step S204.

As in step S400, the parameter determination unit 104 obtains the sizeof an image to be processed (calculation range) (step S1500) Then, thepixel values of pixels included in this calculation range are referredto, and R, G, and B median pixel values P_(R) _(—) _(mid), P_(G) _(—)_(mid), and P_(B) _(—) _(mid) are acquired from all the pixels (stepS1501). Then, R, G, and B pixel values F_(R)(j, i), F_(G)(j, i), andF_(B)(j, i) of the pixel of interest are substituted by the acquiredmedian pixel values P_(R) _(—) _(mid), P_(G) _(—) _(mid), and P_(B) _(—)_(mid) (step S1502)

Note that parameters calculated in step S1500 may be changed todifferent values in correspondence with R, G, and B components.

In the description of the above process, an image is made up of R, G,and B components. However, the above process may be applied to luminanceand color difference components used in JPEG or the like, orcomplementary color components such as C, M, Y, and K components or thelike used as ink colors in a printer or the like.

When many noise components are produced and a narrow processing range isset, the MF process often selects a pixel value that shifts in the noisedirection compared to those around the median pixel value near theoriginal processing region as the median pixel value in the processingregion. On the other hand, when a broad processing range is set, an edgegets into the processing region, and a desired median cannot often beobtained.

Even when the frequency of occurrence of noise is uniform in an image, aregion where noise is conspicuous depends on the color of the region ofinterest. Also, conspicuity of the adverse effect of the noise removalprocess depends on the color of the region of interest, as describedabove.

This embodiment has exemplified the noise removal process using the MFmethod. When the color of the processing region emphasizes noise, theprocessing range is broadened, while when the color of the processingregion emphasizes the adverse effect due to the noise removal effect,the processing range is narrowed down, thus implementing a noise removalprocess that can enhance the noise removal effect and can suppress theadverse effect.

In another method, the MF process may be done only when the pixel valueof interest is isolated compared to those of surrounding pixels. In suchcase, a threshold value used upon determining if the pixel value ofinterest is isolated may be changed in correspondence with the color ofthe processing region.

Fourth Embodiment

The first to third embodiments have exemplified a case wherein variousprocessing parameters are controlled to obtain desired effects andadverse effects of a noise reduction process by utilizing the fact thatthe conspicuity of noise and that of the adverse effect of the noiseremoval process vary with respect to an effective scheme in the noiseremoval process in accordance with the color of the processing region.

FIG. 16 is a block diagram showing the functional arrangement of animage processing apparatus which combines noise removal processes,according to this embodiment. The image processing apparatus shown inFIG. 16 comprises an input terminal 1600, front-end sub noise removalunit 1601, main noise removal unit 1602, and output terminal 1603.

The input terminal 1600 receives an image signal superposed with noise,and the input image is input to the next front-end sub noise removalunit 1601. The front-end sub noise removal unit 1601 executes a noiseremoval process for enhancing the noise removal effect of the main noiseremoval unit 1602. The main noise removal unit 1602 executes a mainnoise removal process. An image that has undergone the noise removalprocess is externally output via the output terminal 1603.

Note that each of the front-end sub noise removal unit 1601 and mainnoise removal unit 1602 comprises the functional arrangement shown inFIG. 1, and their individual noise removal units 105 execute differentnoise removal processes. In this embodiment, the individual noiseremoval unit (which comprises the arrangement shown in FIG. 14) includedin the front-end sub noise removal unit 1601 executes the noise removalprocess using the MF method, and the individual noise removal unit(which comprises the arrangement shown in FIG. 9) included in the mainnoise removal unit 1602 executes the noise removal process using thenoise distribution method.

Note that the flow chart of the noise removal process to be executed bythe image processing apparatus with the above arrangement seriallyexecutes the processes that have been explained in the third and secondembodiments, and a description thereof will be omitted.

FIG. 17A shows an image superposed with noise components, i.e., an imagesuperposed with two different types of noise. Reference numeral 1700denotes visually conspicuous noise, which corresponds to a pixel grouphaving higher or lower pixel values than surrounding pixel values.Reference numeral 1701 denotes spot noise which has a pixel value havinglow correlation with surrounding pixels. On an image input by a digitalcamera or the like, a plurality of different types of noise are oftensuperposed together. The two different types of noise described in thisembodiment correspond to typical types of noise superposed on an imagesensed by a digital camera. FIG. 17B shows the processing resultobtained by applying the noise distribution method executed by the mainnoise removal unit 1602 to the image shown in FIG. 17A.

As has been described in the second embodiment, the noise distributionmethod does not substitute the pixel value of the pixel of interest,when the absolute value of the difference between the pixel values ofthe pixel of interest and selected pixel is large. Since spot noise haslow correlation with surrounding pixel values, it often has a pixelvalue extremely different from surrounding pixel values. When many spotnoise components 1701 are present, as shown in FIG. 17A, the probabilityof pixel value substitution lowers in the noise distribution method. Asa result, the cluster noise 1700 that can be removed by the noisedistribution method cannot often be sufficiently removed, as shown inFIG. 17B. FIG. 17C shows the processing result obtained by applying theMF process to be executed by the front-end sub noise removal unit 1601to the image shown in FIG. 17A. Assume that the MF process of thisembodiment is executed only when the pixel value of the pixel ofinterest is sufficiently different from those of surrounding pixels.

In FIG. 17C, the spot noise 1701 is removed by the MF process, and onlythe cluster noise 1700 remains unremoved. FIG. 17D shows the resultobtained by further applying the noise distribution method to FIG. 17C.In FIG. 17D, the spot noise 1701 is removed, and the cluster noise 1700is distributed, thus implementing effective noise removal.

In this embodiment, the MF process described in the third embodiment isapplied as the front-end sub noise removal unit 1601. Also, the LPFprocess described in the first embodiment similarly has an effect toremove spot noise. For this reason, the LPF process may be applied asthe front-end sub noise removal unit 1601 to obtain the same effect asin this embodiment.

The two different types of noise 1700 and 1701 shown in FIGS. 17A, 17B,17C, and 17D are those which are generated by different generationcauses in case of an image sensed by a digital camera. Also, these noisecomponents have different influences of amplification of theaforementioned image processes and the like. For this reason, a regionwhere each noise is visually conspicuous preferably undergoes arelatively strong process to reliably remove noise. Conversely, a regionwhere the adverse effect of the noise removal process is conspicuouspreferably undergoes a relatively weak process to suppress the adverseeffect. As described above, the conspicuity of the noise and that of theadverse effect of the noise removal process depend on the color of theprocessing region. Furthermore, the conspicuity of the adverse effectdiffers depending on the contents of the noise removal process. For thisreason, when a plurality of different noise removal processes areexecuted, processing parameters are preferably switched for each noiseremoval process in correspondence with the color of a region where noiseto be removed by each noise removal process is conspicuous and that of aregion where the adverse effect of each noise removal process isconspicuous. When the adverse effect is particularly conspicuous,parameters may be determined to skip execution of the process in thefront-end sub noise removal unit 1601 or main noise removal unit 1602.

This embodiment has exemplified the noise removal process using thenoise distribution method as the main noise removal unit, and the LPF orME process as the front-end sub noise removal unit. However, the presentinvention is not limited to such specific embodiment, and a noiseremoval process based on another noise removal method may be executed inadvance. That is, the present invention is effective for a combinationof noise removal methods, which can improve the effect and suppress theadverse effect by such pre-process. Also, this embodiment hasexemplified a case wherein only one front-end sub noise removal processis executed. When a large number of types of noise are superposed, asub-noise removal process may be executed for each noise, and aplurality of different front-end sub noise removal processes may beexecuted in such case.

As described above, upon executing noise removal processes for an imagesuperposed with a plurality of different noise components according tothis embodiment, since the parameters of the removal methods suited torespective noise components are changed in correspondence with the colorof the processing region, respective noise components can be effectivelyremoved. Also, since a plurality of different noise removal methods arecombined, noise components can be effectively removed from an imagesuperposed with a plurality of different noise components.

Fifth Embodiment

The fourth embodiment has exemplified a case wherein the sub noiseremoval process that improves the effect of the main noise removalprocess is executed before the main noise removal process. Thisembodiment will exemplify a case wherein a back-end sub noise removalprocess that suppresses adverse effects is executed in combination afterexecution of the main noise removal process, so as to suppress theadverse effects produced as a result of the main noise removal process.

FIG. 18 is a block diagram showing the functional arrangement of animage processing apparatus which combines noise removal processes,according to this embodiment. The image processing apparatus shown inFIG. 18 comprises an input terminal 1800, main noise removal unit 1801,back-end sub noise removal unit 1802, and output terminal 1803.

The input terminal 1800 receives an image signal superposed with noise,and the input image is input to the next main noise removal unit 1801.The main noise removal unit 1801 executes a main noise removal process.The back-end sub noise removal unit 1802 executes a process thatsuppresses the adverse effects produced as a result of the process ofthe main noise removal unit 1801. An image that has undergone the noiseremoval process is externally output via the output terminal 1803.

Note that each of the main noise removal unit 1801 and back-end subnoise removal unit 1802 comprises the functional arrangement shown inFIG. 1, and their individual noise removal units execute different noiseremoval processes. In this embodiment, the individual noise removal unit(which comprises the arrangement shown in FIG. 9) included in the mainnoise removal unit 1801 executes the noise removal process using thenoise distribution method, and the individual noise removal unit (whichcomprises the arrangement shown in FIG. 3) included in the back-end subnoise removal unit 1802 executes the noise removal process using the LPFmethod.

Note that the flow chart of the noise removal process to be executed bythe image processing apparatus with the above arrangement seriallyexecutes the processes that have been explained in the second and firstembodiments, and a description thereof will be omitted.

FIG. 19A shows an image superposed with noise components. Referencenumeral 1900 denotes a visually conspicuous cluster noise region whichcorresponds to a pixel group having higher pixel values than surroundingpixel values. Reference numeral 1901 denotes a non-noise region otherthan the cluster noise region 1900 in FIG. 19A.

FIG. 19B shows the processing result of the noise distribution methodexecuted by the main noise removal unit 1801 for the image shown in FIG.19A. Reference numeral 1902 denotes noise component pixels as remainingor distributed pixel values of noise components. Reference numeral 1903denotes non-noise component pixels as remaining or distributed pixelvalues of non-noise components. In FIG. 19B, cluster noise componentsare distributed, and are hardly visually conspicuous, thus obtaining acertain noise removal effect.

In recent years, application software or a printer driver often executesan image process such as a color appearance correction process orsaturation up process that changes pixel values. When only the processof the main noise removal unit 1802 is executed, the image shown in FIG.19B is output. When the image shown in FIG. 19B undergoes the imageprocess that changes pixel values, the differences between the noisecomponent pixels 1902 and non-noise component pixels 1903 increase as aresult of the process, thus producing granularity on the entire image.

FIG. 19C shows the result of the LPF process by the back-end sub noiseremoval unit 1803 for the image shown in FIG. 19B. In FIG. 19C, a smoothimage is obtained since it has smaller differences between the noisecomponent pixels 1902 and non-noise component pixels 1903 than those ofthe image shown in FIG. 19B. For this reason, even when the image shownin FIG. 19C undergoes the image process that changes pixel values,production of granularity is suppressed.

Granularity is more likely to be visually recognized with some specificon colors. For example, when a clear sky image suffers granularity evenslightly, the user may feel visually odd. On the other hand, in case ofhuman skin, the surface of which is not completely smooth and has manythree-dimensional patterns, the granularity becomes less visuallyconspicuous. However, when three-dimensional patterns naturally presentas image components are also smoothed as a result of the smoothingprocess, such image appears unnatural.

The conspicuity of granularity varies depending on images of interest.However, clear sky or human skin can be easily detected by detecting thecolor of the region of interest. Hence, when the granularity is visuallyconspicuous in correspondence with the color of the region of interest,parameters are determined to strongly apply the process of the back-endsub noise removal unit 1802, thus suppressing the adverse effect causedby the main noise removal unit 1801. When the parameters that execute aprocess for strongly suppressing the adverse effect are set in theback-end sub noise removal unit 1802, a relatively high threshold valueused to determine pixel value substitution in the noise distributionmethod is set, so that substitution takes place easily, thereby alsoimproving the noise removal effect.

On the other hand, in case of the region of interest in whichgranularity is not visually conspicuous, parameters are determined toweakly apply the process of the back-end sub noise removal unit 1802,thereby suppressing the adverse effect of the whole noise removalprocess. Also, in case of the color region where the differences betweenthe pixel values of the noise component pixels 1902 and non-noisecomponent pixels 1903 are sufficiently small, or they are visuallyinconspicuous, parameters may be determined to cancel the process of theback-end sub noise removal unit 1802. In case of a color region wherenoise itself is inconspicuous, parameters may be determined to weaklyapply or cancel the process of the main noise removal unit 1801.

Note that this embodiment has exemplified a case wherein the LPF processdescribed in the first embodiment is applied as the back-end sub noiseremoval unit 1802. However, the MF process described in the thirdembodiment also has an effect to remove spot noise. For this reason,when the LPF process is applied as the back-end sub noise removal unit1802, the same effect as in this embodiment can be obtained.

This embodiment has exemplified the noise removal process using thenoise distribution method as the main noise removal unit, and the LPF orMF process as the back-end sub noise removal unit. However, the presentinvention is not limited-to such specific embodiment, and the object ofthe present invention can be achieved by various combinations of noiseremoval methods in which one noise removal method causes an adverseeffect, and another noise removal method suppresses the adverse effect.Also, this embodiment has exemplified a case wherein only one back-endsub noise removal process is executed. However, when a plurality ofadverse effects with different characteristics are produced, a pluralityof back-end sub noise removal processes may be used.

As described above, upon executing noise removal processes for an imagesuperposed with noise components according to this embodiment, theadverse effect caused by the noise removal process can be suppressed bycombining a plurality of noise removal methods.

Sixth Embodiment

The fourth embodiment has exemplified a case wherein the effect of themain noise removal process is improved by executing the front-end subnoise removal process before the main noise removal process. The fifthembodiment has exemplified a case wherein the adverse effect of the mainnoise removal process is suppressed by executing the back-end sub noiseremoval process after the main noise removal process. When combinationsof noise removal methods according to the fourth and fifth embodimentsare used in combination, the effects of both the embodiments can besimultaneously obtained.

This embodiment will exemplify a case wherein the combinations of thenoise removal methods described in the fourth and fifth embodiments arefurther combined.

FIG. 20 is a block diagram showing the functional arrangement of animage processing apparatus according to this embodiment. The imageprocessing apparatus shown in FIG. 20 comprises an input terminal 2000,front-end sub noise removal unit 2001, main noise removal unit 2002,back-end sub noise removal unit 2003, and output terminal 2004.

The input terminal 2000 receives an image signal superposed with noise,and the input image is input to the next front-end sub noise removalunit 2001. The front-end sub noise removal unit 2001 executes a noiseremoval process for enhancing the noise removal effect of the main noiseremoval unit 2002. The main noise removal unit 2002 executes a mainnoise removal process. The back-end sub noise removal unit 2003 executesa process for suppressing the adverse effect produced as a result of theprocess of the main noise removal unit 2001. An image that has undergonethe noise removal process is externally output via the output terminal2004.

Note that the flow chart of the noise removal process executed by theimage processing apparatus with the above arrangement can be consideredas serial connection of the processes to be executed by the arrangementsaccording to the fourth and fifth embodiments, since the functionalarrangement of this image processing apparatus is realized by seriallyconnecting the arrangements according to the fourth and fifthembodiments.

The noise removal process according to this embodiment is a combinationof the fourth and fifth embodiments, and can simultaneously obtain theeffects of both the embodiments. That is, the noise removal processaccording to this embodiment can enhance the effect of the main noiseremoval process, and can suppress the adverse effects caused by the mainnoise removal process.

Another Embodiment

The objects of the present invention are also achieved by supplying arecording medium (or storage medium), which records a program code of asoftware program that can implement the functions of the above-mentionedembodiments to the system or apparatus, and reading out and executingthe program code stored in the recording medium by a computer (or a CPUor MPU) of the system or apparatus. In this case, the program codeitself read out from the recording medium implements the functions ofthe above-mentioned embodiments, and the recording medium which storesthe program code constitutes the present invention.

The functions of the above-mentioned embodiments may be implemented notonly by executing the readout program code by the computer but also bysome or all of actual processing operations executed by an operatingsystem (OS) running on the computer on the basis of an instruction ofthe program code.

Furthermore, the functions of the above-mentioned embodiments may beimplemented by some or all of actual processing operations executed by aCPU or the like arranged in a function extension card or a functionextension unit, which is inserted in or connected to the computer, afterthe program code read out from the recording medium is written in amemory of the extension card or unit.

When the present invention is applied to the recording medium, thatrecording medium stores the program codes corresponding to theaforementioned flow charts.

As described above, according to the present invention, not onlyconspicuous noise can be effectively removed from an image superposedwith noise, but also image deterioration can be suppressed.

According to the present invention, the adverse effects caused by thenoise removal process can be effectively suppressed.

As many apparently widely different embodiments of the present inventioncan be made without departing from the spirit and scope thereof, it isto be understood that the invention is not limited to the specificembodiments thereof except as defined in the claims.

1. An image processing apparatus comprising: judgment means for judginga color of an image for each region having a predetermined size in animage superposed with noise; determination means for determining aparameter for a noise removal process in correspondence with the colordetermined by said judgment means; and control means for controllingexecution of the noise removal process in accordance with the parameterdetermined by said determination means.
 2. (canceled)
 3. The apparatusaccording to claim 1, wherein said determination means determinesinformation indicating a size of a region which is to undergo the noiseremoval process, and also determines parameters used in a filteringprocess.
 4. The apparatus according to claim 1, wherein saiddetermination means determines information indicating a size of a regionwhich is to undergo the noise removal process, and also determinesparameters used in a noise distribution method.
 5. The apparatusaccording to claim 1, wherein said control means controls execution of aprocess as one or a combination of a noise reduction process using alow-pass filter, a noise reduction process using a noise distributionmethod, and a noise reduction process using a median filter.
 6. An imageprocessing method comprising: a judgment step of judging a color of animage for each region having a predetermined size in an image superposedwith noise; a determination step of determining a parameter for a noiseremoval process in correspondence with the color determined in thejudgment step; and a control step of controlling execution of the noiseremoval process in accordance with the parameter determined in thedetermination step.
 7. A program for making a computer function as animage processing apparatus of claim
 1. 8. (canceled)
 9. A computerreadable storage medium storing a program of claim
 7. 10. (canceled)